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Abstract 

We prove a new outer bound on the rate-distortion region for the mul- 
titerminal source-coding problem. This bound subsumes the best outer 
bound in the literature and improves upon it strictly in some cases. The 
improved bound enables us to obtain a new, conclusive result for the bi- 
nary erasure version of the "CEO problem." The bound recovers many 
of the converse results that have been established for special cases of the 
problem, including the recent one for the Gaussian version of the CEO 
problem. 

1 Introduction 

In their lauded paper PP, David Slepian and Jack K. Wolf characterize the infor- 
mation rates needed to losslessly communicate two correlated, memoryless in- 
formation sources when these sources are encoded separately. Their well-known 
result states that two discrete sources Yi and Y2 can be losslessly reproduced if 

Ri>H{Yi\Y2) 
R2 > H{Y2\Yi) 
Ri+R2> H{Yi,Y2), 

where Ri is the rate of the encoder observing Yi and R2 is the rate of the en- 
coder observing Y2. Conversely, lossless reproduction is not possible if (i?i,i?2) 
lies outside the closure of this region. See Cover and Thomas |21 Section 14.4] or 
Csiszar and Korner |31 Section 3.1] for precise statements of the result and mod- 
ern proofs. This result is naturally viewed as a multi-source generalization of 
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Figure 1: Separate encoding of correlated sources. 
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Figure 2: A general model. 



the classical result of Shannon 0], which says that, loosely speaking, a discrete 
memoryless source with known law can be losslessly reproduced if and only if 
the data rate exceeds the entropy of the source. Shannon too studied a gener- 
alization of this result, albeit in a different direction. He studied the problem 
of reproducing a source imperfectly, subject to a minimum fidelity constraint, 
and showed that the required rate is given by the well-known rate-distortion 
formula ^ E] . One of the central problems of Shannon theory is to understand 
the limits of source coding for models that combine the two generalizations. 
That is, we seek to determine the rates required to reproduce two correlated 
sources, each subject to a fidelity constraint, when the sources are encoded sep- 
arately (see Fig.0. Determining the set of achievable rates and distortions for 
this setup is often called the multiterminal source-coding problem, even though 
this name suggests a more elaborate network topology. This problem has been 
unsolved for some time. 

The model we consider in this paper is slightly more general and is depicted 
in Fig. 12 Beyond considering an arbitrary number of encoders, L, we also 
allow for a hidden source, Yq, which is not directly observed by any encoder 
or the decoder, and a "side information" source, Y^^i, which is observed by 
the decoder but not by any encoder. We also permit arbitrary functions of the 
sources to be reproduced, in addition to, or in place of, the sources themselves. 
We will therefore use Zi,Z2, etc., to denote the instantaneous estimates instead 
of Yi, ^2, etc., as before. In this paper, we will refer to this more general problem 
as the multiterminal source-coding problem. 
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One might doubt the wisdom of embellishing the model when even the basic 
form shown in Fig. ^ is unsolved. But one of the contributions of this paper 
is to show that far from obscuring the problem, the added generality actually 
illuminates it. Of course, the more general problem is also unsolved. 

Many special cases have been solved, however. For these, the reader is 
referred to the classical papers of Slepian and Wolf mentioned earlier; 
Wyner Ahlswede and Korner [7]; Wyner and Ziv Korner and Mar- 
ton and Gel'fand and Pinsker ^U]; and to the more recent papers of Berger 
and Yeung ^I]; Gastpar ^^l; Oohama ^31; and Prabhakaran, Tse, and Ram- 
chandran ^1]. While all of these papers contain conclusive results, these results 
are established using coding theorems that are tailored to the special cases under 
consideration. 

The solutions to these solved special cases suggest a coding technique for the 
general model The idea is this. Each encoder first quantizes its obser- 

vation as in single-user rate-distortion theory. The quantized processes are then 
losslessly communicated to the decoder using the binning scheme of Cover jl7) . 
The decoder uses the quantized processes to produce the desired estimates. The 
set of rate-distortion vectors that can be achieved using this scheme is described 
in Section |31 This inner bound to the rate-distortion region is tight in all of 
the special cases listed above except that of Korner and Marton Indeed, 
the Korner-Marton problem seems to require a custom coding technique that 
relies on the problem's unique structure. This suggests that the multiterminal 
source-coding problem may not have a classical single-letter solution. 

We attack this problem, therefore, by proving single-letter inner and outer 
bounds on the rate-distortion region. The best inner bound in the literature 
has just been described. The best outer bound, which is due to Berger and 
Tung ^HIj is described in Section|3| In light of the result of Korner and Marton, 
it is clear that the two bounds must not coincide in all cases. This gap cannot 
be entirely attributed to the inner bound, however, as there are instances of the 
problem that can be solved from first principles for which the Berger- Tung outer 
bound is strictly bigger than the true rate-distortion region (see Section IXTI of 
this paper). 

Our aim is to provide an improved outer bound for the problem. We prove 
such a bound in the next section, following a precise formulation of the problem. 
We show that our bound is contained in (i.e., subsumes) the Berger-Tung outer 
bound in Sectional In that section, we also provide several examples for which 
the containment is strict. 

One example is the binary erasure version of the "CEO problem," the general 
version of which was introduced by Berger, Zhang, and Viswanathan ^H]. The 
CEO problem is a special case of the multiterminal source-coding problem in 
which the observed processes Yi , . . . , are conditionally independent given 
the hidden process Iq and in which the decoder (the CEO) is only interested in 
estimating the hidden process^. Berger, Zhang, and Viswanathan characterize 

^This definition is not as restrictive as it might seem. Indeed, any instance of the multi- 
terminal source-coding problem with a single distortion constraint can be transformed into 
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the tradeoff between sum rate and Hamming distortion in tlie liigh-rate and 
many-encoder limit. Gel'fand and Pinsker |10| fiad earlier found the rate region 
in the lossless reproduction case. We consider the problem in which Ig is binary 
and uniform, and the encoders observe Yq through independent binary erasure 
channels. The decoder reproduces Yq subject to a constraint on the "erasure 
distortion" (see Section lX^ or Cover and Thomas [21 p. 370]). For this problem, 
we show that our outer bound is tight in the sum rate for any number of users. 
In contrast, the Berger-Tung outer bound contains points whose sum rate is 
strictly smaller than the optimum. 

In our view, this result is of interest in its own right. The binary erasure CEO 
problem arises naturally in sensor networks in which the sensors occasionally 
"sleep" to conserve energy. This application is described in Section IX^ The 
result also provides an example for which the binning-based coding scheme 
mentioned earlier is optimum. Finally, this is one of relatively few conclusive 
results for the multiterminal source-coding problem in general, and the CEO 
problem in particular. These problems are considered sufficiently difficult that 
it is worth reporting solutions to special cases. 

One of the few other conclusive results available is for the Gaussian version of 
the CEO problem, which was first studied by Viswanathan and Berger Here 
the encoders observe a hidden Gaussian source through independent Gaussian 
additive-noise channels. The distortion measure is expected squared error. The 
rate-distortion region for this problem was recently found by Oohama [201 ^| 
and independently by Prabhakaran, Tse, and Ramchandran We show that 
the converse result of these four authors can be recovered from our single-letter 
outer bound, while the Berger-Tung outer bound contains points that lie outside 
the true rate-distortion region. 

The converse results used to solve all of the other special cases mentioned 
so far are also consequences of our bound. This is discussed in Section ^ Our 
outer bound therefore serves to unify most of what is known about the nonex- 
istence of multiuser source codes. This unification is noteworthy in the case of 
Oohama |13] and Prabhakaran, Tse, and Ramchandran |14| because the connec- 
tion between their remarkable converse result and the classical discrete results in 
this area is not immediately apparent. As we will see, subject to some technical 
caveats, most of the key results in multiterminal source coding can be recovered 
by combining the general inner bound described earlier with the outer bound 
described next. 

2 Formulation and Main Result 

We work exclusively in discrete time. We use uppercase letters to denote ran- 
dom variables and vectors, lowercase letters to denote their realizations, and 
script letters to denote their ranges. Let {Yf^{t), 17(i), • ■ • , Y£{t), IT+i 

an instance of the CEO problem without changing the rate-distortion region by lumping 
Yi, . . . , Yl into Yq and redefining the distortion measure as needed. Nonetheless, it defines a 
useful special case. 
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Figure 3: Notation for the encoding and decoding rules. 



be a vector-valued, finite-alphabet memoryless source. For A C {1, . . . we 
denote (V(0){feA} hyY\{t). If A = {1, . . . , L}, we write this simply as Y"(t). 
In this context, the set A'^ should be interpreted as {1, . . . , L}\A rather than 
{0, . . . , L \}\A. When A = {£}, we shall write Yf{t) and r/](t) in place of 
and Y{fi,c(t), respectively. Also, we use Y[-{ti : to denote {i^" (0}t=ti : 
Yp to denote r/(l : n), and Yf"(t^) to denote 

(r,"(l), . . . , Yl\t - l),Yl\t + 1), . . . , y/(n)). 

Similar notation will be used for other vectors that appear later. 

The notation for the encoding and decoding rules is shown in Fig. |21 For 
each I in {1, . . . , L}, encoder £ observes Y^", then employs a mapping 



to convey information about it to the decoder. The decoder observes Y^+i ^^"^ 
uses it and the received messages to estimate K functions of the vector-valued 
source according to the mappings 



Z]^ for k 



1,- 



We assume that K distortion measures dk : Ilfclo^ x 2^ R+ are given. 

We mention at this point that while the generality of this setup will be 
useful later when studying examples, it is not needed to appreciate the bounding 
technique itself. The reader is welcome to focus on the basic model shown in 
Fig. n for that purpose. 



Definition 1 The rate- distortion vector 



(R,D) = (i?i,i?2 



is achievable if there exists a block length n, encoders , and a decoder 



, Rl, Di,D2, . . . , Dk) 
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such that^ 



Ri>- logA^"^ for all i, and 



Dk > E 



n 



n 
t=i 



for all k. 



(1) 



Let TZT)^ be the set of achievable rate-distortion vectors. Its closure, TZD^, is 
called the rate-distortion region. 

We will sometimes be concerned with projections of the rate-distortion region. 
We denote these by, for example, 7?.I?* n {Ri = 0}, meaning 

{{R2,---,rl,di,...,dk) ■■ {o,r2,...,rl,di,...,dk) ewn:} . 

In this paper, we view lossless compression as a limit of lossy compression 
with the distortion tending to zero. More precisely, if we wish to reproduce Yi 
losslessly, we will set, say, Zi = yi with di equal to Hamming distance, and 
then examine ^l'D^,^]{Dl = 0}. This convention and Definition ^ together yield 
a notion of lossless compression that is weaker than the one traditionally used. 
It is common instead to require that for all sufficiently large block lengths, there 
exists a code for which the probability of correctly reproducing the entire vector 
Y" is arbitrarily close to 1. But a weaker notion is desirable here since we are 
proving an outer bound or "converse" result. 

To state our result, let Yq, . . . , Yl+i be generic random variables with the 
distribution of the source at a single time. Let To denote the set of finite- 
alphabet random variables 7 = (Ui, . . . , Ul, Zi, . . . , Z^, W, T) satisfying 

(i) (W,T) is independent of {Yo,Y,Yl+i), 

{li) Ui ^ {Ye, W, T) ^ (Fo, Y^c, F^+i, U^,,), shorthand for "C/^ [Yi, W, T) and 
(Yb: Yfc, yj^_i_j, U^c) form a Markov chain in this order", for all I, and 

(m) {Yo,Y,W)^{\J,Yl+i,T)^Z. 

It is straightforward to verify that To is precisely the set of finite-alphabet 
random variables (C/i, . . . , Ul, Z\, . . . , Zk, W, T) whose joint distribution with 
(Fq, Y,yL+i) factors as 

L 

P{ya, y, yL+i, u, z, w, t) = p{yo, y, yL+i)p{w, t) Wp{ut\yt, w, t)p{z\u, yL+i,t). 

t=\ 

This description is helpful in that it suggests a parametrization of the space To- 
Let X denote the set of finite-alphabet random variables X with the property 
that Y\, . . . ,Yl are conditionally independent given (X, Y^+i). Note that x is 
nonempty since it contains, e.g., X = (Yi, . . . , Yi_i). 

^AU logarithms and exponentiations in this paper have base e. 
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There are many ways of coupling a given X in x and 7 in To . In this paper, 
we shah only consider the unique couphng for which X ^ {Yo,Y,Yl+i) ^ 7, 
which we call the Markov coupling. Whenever the joint distribution of X, 
(Fo, Y, Yl+i), and 7 arises, we assume that this coupling is in effect. 

It is evident from the definition of x that there is considerable latitude in 
choosing how X depends on Yq. This is because the sole constraint on the choice 
of X only depends on the joint distribution of X and {Y,Yl^i). But as the 
following definition makes clear, this freedom is inconsequential since our outer 
bound only depends on the distributions of {Yq, Y, y^+i, 7) and {X, Y, Yl+i,j) 
separately. 

Definition 2 Let 

nVo{X,j) = J (R,D) : ^i?, >/(X;UA|U^c,rL+i,T) 

+ J2 HYi; Ui\X, Yl+i,W, T) for allA^{l,..., L}, 
and Dk > E[dkiYo,Y,YL+i, Zk)] for all fcj. 

Then define 

The first theorem is our main result. 

Theorem 1 The rate- distortion region is contained in TZVo- In fact, 7?. 2?* C 

Proof. It suffices to show the second statement. Suppose (R, D) is achiev- 
able. Let /}"\ . . . , /I""* be encoders and {(p^j^\ . . . , (/J^f "*) a decoder satisfying 
Take any X in x and augment the sample space to include X" so that 

{X-{t),Y^\t),Y-{t),Yf:^,{t)) 

is independent over t. Next let T be uniformly distributed over {!,..., n}, 
independent of X", Y^, Y", and r/^^.^. Then define 

X = X"{T) 

Yi = y/(r) for each ^ in {0, . . . , L + 1} 

Ui = [.ft\Yn,X"{l : T - l),YE+,ir)) for each i 

Zk = Zl!{T) for each k 
W = {X"{T^),Y£^,{r)). 
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It can be verified that 7 = (U, Z, W, T) is in Fq and that, together with Yq, Y, 
Yl+1, and X, it satisfies the Markov couphng. It suffices to show that (R, D) 
is in 7?.I>o(X, 7). First, note that imphcs 

Dk > E[dk{Y,-{T),Y"{T),Yl\,{T),Zl\T))] for all fc. 



I.e., 



Dk > E[dk{Yo,Y,YL+uZk)] for all k. 
Second, let A C {1, . . . , L}. Then by the cardinality bound on entropy. 



nJ2R,>HUf^-\Yn 
Since conditioning reduces entropy, this implies 



By the chain rule for mutual information. 



eeA' 

ft\Yn 



L+l 



L+l 



Y, 



L+l 



liX 
1 1 Y 
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(2) 



(3) 



Applying the chain rule again gives 

n , 

Consider next the second term on the right-hand side of JSJ- Since X E Xi 

j2^{Y";t\Yn 



0- 
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Applying the chain rule once more gives 

/(y/;/f)(V) 



t=i 



But 



+ I{Yp{t);Yl\l:t-l)\X-,Y£^,) 

= i(Ynt)-jl''\Yn\x'',yE+i) 

+ i{Yi\t)-Yp{\ : )(r,"),x",y2Vi) > 

and the second term on the left-hand side is zero. Thus 



(y/W;/f)(y/) 



Substituting the results of these various calculations into (01 gives 



1 " r / \ 

+ 5] / (y,"(t); /(")(y,") yr+i W, 



(4) 



If A'' is nonempty, this can be rewritten as 

5] i?£ > /(X" (T) ; I U^. , y^\i (T) , T) 

+ ^ /(y,"(r); c/,|x"(r), y£Vi(T), y/'+iiT^), r) 

= /(X; UaIUac, y^+i, T) + ^ /(y^; C/,|X, yL+i, M^, T). 
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The case A = {1, . . . , L} is handled separately. In this case, observe that 

= /(x"(t);(/(")(r, 



eeA V / eeA' 



+ i{x-{t)-x-{\ : t ~ 1), nVi(OI>T+i W) 

Substituting this into |0J and proceeding as in the A"^ 7^ case completes the 
proof. □ 

It is worth noting that the proof uses classical techniques. Most of the ma- 
nipulations in the latter part of the proof can be viewed as versions of the chain 
rule for mutual information. Since this chain rule holds in abstract spaces |21l 
(3.6.6)], the proof can be readily extended to more general alphabets. 

The key step in the proof is the introduction of X" in Unlike the other 
auxiliary random variables, does not represent a component of the code. 
Rather, it is used to aid the analysis by inducing conditional independence 
among the messages sent by the encoders. This technique of augmenting the 
source to induce conditional independence was pioneered by Ozarow |22) . who 
used it to solve the Gaussian two-descriptions problem. Wang and Viswanath [23| 
used it to determine the sum rate of the Gaussian vector multiple-descriptions 
problem with individual and central decoders. It was also used by Wagner, 
Tavildar, and Viswanath to solve the Gaussian two-terminal source-coding 
problem. A step that is similar to ^ appeared in Gel'fand and Pinsker [TUI 
and in later papers on the Gaussian CEO problem although in these 

works X" is part of the source, so no augmentation is involved. 

The significance of conditional independence has long been known in the 
related field of distributed detection (e.g., |55]). Given the similarity between 
distributed detection and the multiterminal source-coding problem, one expects 
conditional independence to play a significant role here as well. Indeed, most 
conclusive results for the multiterminal source-coding problem require a condi- 
tional independence assumption |1(JI 1181 1121 1131 [Tlj . The motivation for intro- 
ducing X" is that it allows one to apply the approach used in these works to 
problems that lack conditional independence. 

We do not consider the problem of computing T^Vq in this paper. Note that 
we have not specified the alphabet sizes of the auxiliary random variables U, W ^ 
and T . As such, the outer bound provided by Theorem ^ is not computable |31 
p. 259] in the present form. One might question the utility of an outer bound 
that cannot be computed. The remainder of the paper, however, will show 
that the bound is still useful as a theoretical tool. In addition, cardinality 
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bounds might be found later, although obtaining such bounds appears to be 
more difficult in this case than for related bounds. 

It should be mentioned that the time-sharing variable T is unnecessary; 
it can be absorbed into the other variables. We have included it to ease the 
comparison with existing inner and outer bounds, to which we turn next. 

3 Relation to Existing Bounds 

The coding scheme described in the introduction gives rise to the following inner 
bound on the rate-distortion region. 

Definition 3 Let Tf^ denote the set of finite- alphabet random variables 
7 ([/i, . . . , C/l, Zi, . . . , Zk,T) 

satisfying 

(i) T is independent o/ (Fq, Y, Y^+i), 

^ii; Ui ^ {Yi,T) ^ (Fo, Y,c,yi+i,U£c) for all i, and 
(ui) iYo,Y)^{U,YL+i,T)^Z. 
Then define 

TlVf^i-f) = I (R,D) : ^i?, > /(YA;U^|U^c,yi+i,T) for all A, 
and Dk > E[dk{Yo,Y,YL+uZk)] for all fcj. 

Finally, let 

nvr= U 7^pr(7). 



Proposition 1 ((1111111) TlVf^ C 7^X>^. 

In Appendix El we show that 7^Pf ^ is in fact closed. We call UVf^ the 
Bergcr-Tung jl5l inner bound, since although these authors prove a bound 
that is less general than the one given here, their proof can be extended to prove 
Proposition n See Chen et al. [201 or Gastpar [T^ for recent sketches of the 
proof that accommodate some of the generalizations included here. 

To understand the difference between TZVf^ and TZVo, suppose that 

(u,z,iy,T) 
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is in To and W is deterministic. Then (U, Z,T) is in Ff-^, and for all A C 
{1, . . . , L} and all X € Xj 



liX; U^IU^e, r^+i, T) + nyf, Ui\X, Yl+1, W, T) 



= I{X- U^IU^e, y^+i, T) + /(Y^; U^IU^e, X, Yl+i,W, T) 
= I{X- UaIU^c, y^+i, T) + /(Ya; U^IU^e, X, y^+i, T) 
= /(X,YA;UA|UAc,yi+i,T) 
= /(YA;UA|UAc,yi+i,T). 



Thus 



7^I?o(x,u,z,^^,^) = 7^pf^ (u,z,t). 



(5) 



Conversely, if (U, Z, T) is in Tf^, then for any deterministic W, (U, Z, W, T) is 
in To and © holds for any X. It follows that 7^Pf ^ is equal to TlVo with W 
restricted to be deterministic in the definition of Fq. 

In particular, to obtain coincident inner and outer bounds, it suffices to show 
that restricting W to be deterministic in the definition of Fq docs not reduce 
TlVo- We will see later how this can be accomplished in several examples. Of 
course, it is not possible for the problem solved by Korner and Marton [^, since 
they show that the inner bound is not tight in that case. 

The best outer bound in the literature is the following. 

Definition 4 Let Tf^ denote the set of finite- alphabet random variables 7 = 
(U, Z, T) satisfying 



(i) T is independent of {Yq,'Y,Yl^i), 



(ii) Ut ^ {Yi,T) ^ (Fo, Y^c,yi+i) for all I, and 



fiiij (yo,Y)^(u,yi+i,r)^z. 



Then let 




and Dk > E[dk{Yo,Y, Yl+uZu)] for all k 



Finally, let 



nv, 



o 



Proposition 2 (HSHTn]) 7^X'* C TIV, 



o 
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As with the inner bound, Berger and Tung prove the result for a model 
that is more restrictive than the one considered here, but their proof can be 
extended to this setup (c.f. EnHEl)- The difference between UVf^ and Tepf^ 
is that condition (ji) has been weakened in the latter. We next show that the 
Berger- Tung outer bound is subsumed by the one in the previous section. 

Proposition 3 UVo C 7^X'f^. 

Proof. First observe that for any (U, Z, W^T) in Fq, 

Ut ^ VF, T) ^ (Fo, Y,.,Yl+i) for each L 
Since (yo,Y,c,yi+i) ^ {Y,,T) ^ W, it holds 

Ui ^ {Ye,T) ^ iYo,Ye^,YL+i) for each i. 
Thus (U, Z,T) is in F^^ and in particular, 

7^X'o(Y,U,Z, VF,T) = 7^X'f'^(U,Z,T). 

It follows that 

TZVo c IJ nVaiY,j) c U 7^I?f^(7) = nv^^. 

□ 

The proof reveals that TZVo improves upon TZV^^ in two ways. The first 
is that TZVo allows for optimization over X while TZV^^ effectively requires 
the choice X ~ Y. The second is that To is "smaller" than F^^ in the sense 
that if (U, Z, W, T) is in then (U, Z, T) is in Ff^. The balance of this 
section is devoted to showing that these improvements make the containment 
in PropositionOstrict in some cases. As the reader will see, the former difference 
is entirely responsible for the gap that we expose between the two bounds in 
our examples. We hasten to add, however, that the latter improvement is not 
an empty one in that Anantharam and Borkar [27] have shown that there can 
exist a (U, Z,T) in F^-^ with the property that there docs not exist a W such 
that (U, Z, T) is in Fq. It is interesting to note that the Anantharam-Borkar 
example arose independently of this work in the context of distributed stochastic 
control. 

We will exhibit three examples for which TZT)^^ strictly contains TZVo- The 
first is rather contrived and can be solved from first principles. It is included to 
illustrate the difference between the two bounds. 

3.1 Toy Example 

Let Yii, Yi2, 121, and Y22 be independent and identically distributed (i.i.d.) 
random variables, uniformly distributed over {0, 1}. Consider two encoders 
(L = 2) with Yi = (yii,yi2) and Y2 = (^21,^22) (there is no hidden source 
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or side information in this example). We have a single distortion constraint 
{K = 1) with Zi = {0, 1}2 and 



rfi(Y,Zi) 



if Zi = (^11,^21) or Zi = (yi2, F22) 

1 otherwise. 



In words, the decoder attempts to guess either the first or the second coordinate 
of both encoders' observations. It incurs a distortion of zero if it guesses correctly 
the same coordinate of the two sources and one otherwise. Note that the decoder 
need not declare which coordinate it is attempting to guess. 

Proposition 4 For this problem, 



nVor\{Di=Q] C {(i?i,i?2) : Ri > log2,i?2 >log2}. 

Proof. Suppose (i?i,i?2,e) is in TZVo, and e < 1/2. Observe that since Yi and 
Y2 are independent, deterministic random variables are in x- Thus there exists 
7 in To such that 

e>E[di{Y,Zi)] 
Ri > I{Yi;Ui\W,T) 
R2>I{Y2;U2\W,T)- 
By condition (ii) defining Fq, 

Ui^iYi,W,T)^{Y2,U2). (6) 

Since Y2 is independent of (Yi, W,T) in this example, Y2 must be independent 
of {Yi,Ui,W,T). Thus 

I{Yi;Ui\W,T) ^ I{Yi;Ui\ W, T,Y2i ^ Y22 ) . (7) 

Likewise, Yi is independent of {Y2,U2,W,T) and hence given {W,T), Yi is in- 
dependent of {Y2,U2). This observation combined with @ implies (Yi,t/i) 
{W,T) ^ {Y2,U2). In particular, Fi ^ {Ui,W,T) ^ (l2,C/2). By condition 
(m) defining To, Yi <^ {Y2,Ui,U2,W,T) ^ (Y2,Zi). These last two chains 
imply that 

Yi <^ iUi,W,T) <^ (Y2,Zi). 

Thus conditioned on {W,T) and the event {Y21 7^ Y22}, we have Yi ^ [/i 
(Y2,Zi). It follows that 

/(Yi;C/i|I^,r,Y2i ^ Y22) 
>/(Yi;Y2,Zi|M^,T,Y2i ^ Y22) 
= /( Yi ; Y2 , , rfi (Y, Zi ) I W, T, Y21 ^ Y22 ) 

- /(Yi :di{Y,Zi)\W, T, Y2 , Zi , Y21 ^ Y22) 

> H{Yi\W, T, Y21 ^ Y22) - H{Yi\Y2, Zi, di (Y, Zi), M^, T, Y21 ^ Y22) 

- (di ( Y, Zi ) I M^, T, Y2 , Zi , Y21 ^ Y22 ) , 



14 



since di(Y,Zi) is a function of Y and Zi. Next, observe that on the events 
{Y21 ^ Y22} and {di{Y , Zi) = 0}, Y2 and Zi together must reveal one of the 
two bits of Yi . Thus 

H{Yi\Y2,Zi,diiY, Zi) 0, W,T, Y21 +Y22) < log 2. 

Continuing our chain of inequalities, 

I{Yr.Vx\W,T,Y2x+Y22) 
>21og2-log2-Pr(di(Y,Zi) = 01^21 ^1^22) 

- (2 log 2) • Pr(di(Y, Zi) = 11^21 ^ Y22) 

-H{di{Y,Zi)\Y2i^Y22) 
> log 2 - (2 log 2) • Pr(di(Y, Zi) = l\Y2i ^ Y22) (8) 

-i/(dl(Y,Zi)|y21 ^>22). 

Now 

]^H{di{Y,Zi)\Y2i^Y22) + \H{di{Y,Zi)\Y2i^Y22) 

= H{di{Y, Zi)|l(y2i - Y22)) < i?(di(Y, Zi)) < /i(e), 

where, here and throughout, h{-) is the binary entropy function with natural 
logarithms. Wc conclude that 

H{di{Y,Z,)\Y2,^Y22)<2h{e). 

Similarly, 

Pr(di(Y,Zi) = 11^21 ^y22)<2e. 
Substituting these two observations into and recalling Q yields 

I{Yi ■,Ui\W,T)> log 2 - 4e log 2 - 2h{e). 

By symmetry, /(I2; U2\W, T) must satisfy the same inequality. This implies the 
desired conclusion. □ 
It is easy to see that the point (log 2, log 2, 0) is achievable. Using rate log 2, 
each encoder can send, say, the first coordinate of its observation. The decoder 
can then realize zero distortion by repeating the two bits it receives. This fact 
and the above proposition together imply 



7^Po n {Di = 0} = nv^ n {Di = o} = {(i?i, i?2) : Ri > log 2, R2 > log 2}. 

In particular, TZVo is tight in the zero-distortion limit. In contrast, we show 
next that the Berger-Tung outer bound is not. 



Propositions T/ie poinf ((3/4) log2, (3/4) log2, 0) is contained in TZV ^ 



BT 
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Proof. Let the random variable W be uniformly distributed over {1,2}, and let 
Ui = Yiw and U2 = Y2w- Let Zi = (C/i,[/2). It is straightforward to verify 
that {1/1,1/2, Zi) is in F^-^ (the time-sharing random variable T is unneeded 
and can be taken to be constant). Next note that E[di{Y , Zi)] = 0. Finally, 
one can compute 

/(Y;C/i,C/2) = Jlog2 

and 

J(Y;;7i) = ilog2. 

This implies that 

/(Y;[/i|[/2) =/(Y;(72|C/i) = Jlog2 > ^log2. 
The conclusion follows. □ 



3.2 Binary Erasure CEO Problem 

Here Yq is uniformly distributed over {—1,1}, and Yi = NcYq for ^ in {1, . . . , L}, 
where Ni,...,Nl are i.i.d. with < Pr(7Vi = 0) = p < 1 and Pr(iVi = 1) = 
1 —p. Let Zi ~ {—1, 0, 1}. We will assume that there is no side information and 
that the decoder is only interested in reproducing the hidden process Yq. We 
measure the fidelity of its reproduction using a family of distortion measures, 
{di}\>o, where 

fO ifyo = ^i 
d^{YQ,Y,Zi) = h ifZi=0 
[ A otherwise. 

Wc arc particularly interested in the large-A limit. In this regime, approxi- 
mates the "erasure distortion measure" [3 p. 370], 

r if Fo = Zi 
dr(^o,Y,Zi) = i 1 ifZi = 
I 00 otherwise. 

Wc use a finite approximation because an infinite distortion measure causes 
difficulties in the proof of the Berger-Tung inner bound. 

This example is motivated by the following problem arising in energy-limited 
sensor networks. We seek to monitor a remote source, Yq. To this end, we 
deploy an array of sensors, each of which is capable of observing the source with 
negligible probability of error. To lengthen the lifetime of the network, each 
sensor spends a fraction p of the time in a low-power "sleep" state. We assume 
that the sensors cycle between the awake and sleep states independently of each 
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other and on a faster time scale than the sampHng; at each discrete time, each 
sensor sleeps with probability p, independently of the other sensors and the 
past. Sensors do not make any observations while they are asleep, resulting in 
erasures. We permit the coding process to introduce additional erasures, but 
not errors, yielding the erasure distortion measure. What sum rate is required 
in order for the decoder to reproduce a fraction 1 — £> of the Fq" variables while 
almost never making an error? Of course, D must satisfy D > p^. 
Define 

7^,(Z?,A) =inf l^i?, : , . . . , i?^, 2?) € ^(A) | , 



where TZ'Di.^X) is the rate-distortion region when the distortion measure is d^. 
We define TZoiD, A) and 7^f'^(D, A) analogously. 

In Appendix ^ we show that if < -D < 1 , then 



lim nf^iD, A) < (1 - D) log2 + L 

A— ^oo 



(9) 



In Appendix^ we show that the quantity on the right-hand side is also a lower 
bound to limx^oo'Ti'oiD, X). Hence it must equal lhnx—fooT^*{D,X). That is, 
the improved outer bound and the Berger-Tung inner bound together yield a 
conclusive result for the sum rate of the binary erasure CEO problem. Evidently 
this problem was previously unsolved. In Appendix [HI we show that TZD^'^ 
contains points with a strictly smaller sum rate in general. Fig. 01 shows the 
correct sum rate for p = 0.5 and several values of L. 



3.3 Gaussian CEO Problem [TUl I251ITSIIT11 

We turn to a continuous example. Here Yq, . . . ,Yl arc jointly Gaussian and 
Yi, . . . ,Yl arc conditionally independent given Yq . For £ > 1 , let us write 
Yg = Yq + Ng, where Yq, Ni, . . . , Nl are mutually independent and 

E [Nj] = cr^ > for all 

We will denote the variance of Yq by . Again there is no side information, 
and the decoder is only interested in reproducing the hidden process Yq, 

di(ro,Y,Zi) = {YQ-Z^f. 

The rate-distortion region for this problem was recently found by Oohama |2U[ 
[T5] and Prabhakaran, Tse, and Ramchandran ^1]. The two proofs are nearly 
the same, and build on earlier work of Oohama |28j . The primary contribution 
is the converse result, which makes heavy use of the entropy power inequality 12 
Theorem 16.6.3]. The Berger-Tung inner bound is used for achievability. 

It is straightforward to extend Theorem ^ to this continuous setting. A 
statement of the continuous version is given in Appendix ^ where we also use 
the techniques of Oohama ^21 a-nd Prabhakaran, Tse, and Ramchandran [T^ 
to prove the following. 
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Tolerable Fraction of Erasures {D) 



Figure 4: Sum rate for the binary erasure CEO problem with p = 1/2. 



Proposition 6 For the Gaussian CEO problem, 



nVo C <^ {Ri,...,Rl,D) G 



there exists (ri, . . . , r^,) G '■ for all A, 



1 




Y^r, 



(10) 



where log"'' x = max(loga;, 0). Since this expression equals 7?.I?^ |13j . we con- 
clude that TZDo is tight in this example. It also follows that the converse result 
of Oohama |E1 and Prabhakaran, Tse, and Ramchandran ^31 is a consequence 
of the outer bound provided in this paper. This does not imply, however, that 
the task of proving the converse result is made any easier by our bound. In fact, 
comparing Appendix IdI to the original works shows that proving Proposition |B1 
is as formidable a task as proving the converse result unaided. But this is still 
an improvement over the Berger-Tung outer bound, the closure of which we 
show in Appendix El contains points outside the rate-distortion region. 

We end this section by mentioning that Oohama's ^31 converse is actually 
more general than the result described here, in that Oohama permits one of the 
encoders to make noise- free observations (i.e., = 0). Comparing Oohama's 
proof to Appendix ^ shows that the outer bound supplied in this paper also 
recovers this more general result. 
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4 Recovery of Discrete Converse Results 



Having seen that the new outer bound recovers the converse of Oohama jl3j and 
Prabhakaran, Tse, and Ramchandran ^l] for the Gaussian CEO problem, we 
show in this final section that it also recovers the converse results for the discrete 
problems of Slepian and Wolf P , Wyner [S| , Ahlswede and Korner [7j , Wyner 
and Ziv |S], Gel'fand and Pinsker JUj, Berger and Yeung ^T], and Gastpar |12) . 
The outer bound also recovers the converse result for the problem studied by 
Korner and Marton |U] , although the proof of this fact is not as interesting. We 
shall therefore focus on the others. To recover these converse results, we shall 
use the following conclusive result for a special case of the problem. 

Suppose that there exists a function g : 3^o 3^o such that Yi , . . . , Yf, are 
conditionally independent given (g(yo), ^L+i)- Also let Z\ = J^q and 



if Zi = Yo 

1 otherwise. 



diiYo,Y,YL+i,Z,) 

We make no other assumptions about the problem. We would like to character- 
ize the set TZVi, n {Di =0}. In words, conditioned on the side information and 
some function of the hidden variable, the observations are independent, and the 
hidden variable must be reproduced losslessly. Note that TZV-i, n {Di = 0} will 
be empty unless H{Yo\Y,Yl+i) = 0. Gel'fand and Pinsker ^U] refer to this 
condition as "completeness of observations." 

Proposition 7 For this problem, 



nVo n {Di 0} = nvf^ n {Di = o} = nv^ n {Di = o} (ii) 

= U 7^2?^(7) n {7^1 = 0}. (12) 



Proof. To show (tTlTl . it suffices to show that TZVo n {Di = 0} is contained in 



nvf^ n {Di = 0}. Suppose (R, e, i:>2, . . . , Dk) is a point in UV^ and e < 1/2. 
By choosing X = giYo) in Definition |21 we see that there exists (U, Z, W, T) in 
To such that 

Pr(Zi ^ Fo) < e 
E[dk{Yo,Y, Fl+i, Zk)] < Dk for all fc > 2, 

and for all A, 

Y.^'^ ^^9{Yo)-, U^IU^c, yi+i,f ) + nYe; Ui\giYo), Yl+uW, f). 

Now 

/(5(ro); UaIU^c, Fi+i, f ) = H{g{Yo)\\JM,YL+i,f) 

~H{g{Ya)\{J,YL+i,f) 
> H{g{Y^)\\JA.,YL+i,W,f) ^ H{g{Yo)\g{Z^)), 



19 



where we have used the fact that 

giYo) ^Yo^ (U, YL+i,f) ^ Z, ^ g{Zi). 
By Fano's incquahty O Lemma 1.3.8], 

H{g{Yo)\g{Z^))<h{e) + e\og{\yo\). 

Thus 

/(.g(ro);U^|U^<=,lL+i,f) > H{g{Yo)\\5A^,YL+i,W,f) - h{e) - elogdJ^ol) 

> I(g{Yo);UA\VA^,YL+i,W,f) 
-/i(e)-elog(|3^o|). 

It follows that 

Y,[Ri + H^) + ^^og{\yo\)]>I{g{Yo);lJA\VA^,YL+i,W,f) 

eeA 

+ Y,IiYf,Ue\g{Yo),YL+i,W,f) 

eeA 

= I{g{Yo);VA\UA^,YL+i,W,f) 

+ IiYA;UA\UA^,g{Yo),YL+i,W,f) 
= HgiYo), Ya; \Ja\Va^,Yl+i,W, f) 

>I{YA-,\JA\\JA^.YL+uW,f). 

If we now define T = {W, f), it is evident that (U, Z, T) is in Vf^ and the point 

[Ri + h{e) + e log(|3^o|), • ■ • , i?L + h{e) + e log(|3^o|), e, i?2, • ■ • , Dk) 
is in TZVf^. This implies that 

nv^r\{Di = 0} c nvf^ n {Di = o}, 

which proves Hll() . To prove H12|l . it suffices to show that TZVf^ is closed. This 
is shown in Appendix IfI □ 
The differences between this result and that of Gel'fand and Pinsker [Tllj 
are numerous but minor. The most visible differences are that Gel'fand and 
Pinsker's model does not allow for side information at the decoder or distortion 
constraints beyond the one on Yq. Indeed, the region given here reduces to theirs 
when these extensions are ignored. Thus this result seems to be a generalization 
of theirs, albeit a trivial one since their proof can be modified to handle these 
extensions. A closer comparison, however, reveals that they define the rate 
region more stringently than we do here. Thus, our result does not recover 
theirs, strictly speaking, although it does recover the converse component of 
their result since our definitions are weaker. 
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The reason for including side information and additional distortion con- 
straints in the model is that they enable us to also recover the converse results 
for the other problems mentioned earlier. For instance, Gastpar ^21 considers 
the problem of reproducing the observations individually, subject to separate 
distortion constraints, under the assumption that the decoder is provided with 
side information that makes the observations conditionally independent. His 
converse result can be recovered by setting Yq = Yl+i- It is easily verified that, 
under this condition, our region coincides with his. The classical Wyner-Ziv 
problem |S] can be viewed as Gastpar's problem with a single encoder {L = 1). 
So that converse result is recovered too. 

Berger and Yeung solve the two-encoder problem in which the obser- 
vations are to be reproduced individually, with at least one of the two being 
reproduced losslessly. In our notation, this corresponds to setting L = 2 and 
Yi = 10- Note that our conditional independence assumption necessarily holds 
in this case. 

To see that under these assumptions, our region reduces to theirs, suppose 
i?2, £'2) e nvf^ij) n {Di = 0} for some 7 e Then 

Ri>IiYi;Ui\U2,T)=H{Yi\U2,T). (13) 

Also, 

i?2 > / ( Y2 ; t/2 1 C/i , T) > 7(^2 ; C/2 1 C/i , Fi , T) 

= I{Y2;U2\Yi,T), (14) 

where we have used the fact that 

U2^(Y2,Ui,T)^Yi 

(see Cover and Thomas |21 p. 33]). Finally, 

Ri+R2>IiYi,Y2;Ui,U2\T) 

> /(Fi ; C/i , C/2 IT) + I{Y2 ; C/i , C/2 iFi , T) 

>H{Yi) + I{Y2;U2\Yi,T). (15) 

It is now evident that the two regions arc identical (c.f. ^2 P- 230]). Thus 
the converse result of Berger and Yeung is a consequence of the outer bound 
provided here. 

The classical problem of source coding with side information jS] [7j can be 
viewed as a special case of the Berger- Yeung problem in which D2 exceeds 
the maximum value of ^2, the distortion measure for Y2. Berger and Yeung 
demonstrate how, under this assumption, the region described above reduces 
to the one given by Wyner [H] and Ahlswede and Korner [7|. Ipso facto, the 
converse result for this problem is also recovered. 

This paper ends the way it began, with the result of Slepian and Wolf 
Here the aim is to losslessly reproduce all of the observations. For two encoders 
(L = 2), this can be viewed as a special case of the problem of Berger and 
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Yeung. These authors show how the region described in Eqs. ((T^ - l(T^ reduces 
to the one given at the beginning of the paper. The result for more than two 
encoders can be viewed as a special case of Proposition [7| in which Yq — Y. In 
this case, if R € TZVo n {Di = 0}, then for any A, 

^i?, >/(Y^;U^|Ua<=,T) 

eeA 

= iJ(Y^|UA=,T) 
>H{Ya\Ya^,T), 

since Ya ^ {Ya<=,T) {JJ a-'tT). Now Y is independent of T, so 

Y,Re>H{YA\YA^), 

eeA 

which is the well-known rate region for this problem. Thus the converse of 
Slepian and Wolf is also recovered. For this result, as with the others, our outer 
bound dispenses with the need to prove a custom converse coding theorem. In 
fact, Proposition[7|can be viewed as unifying all of the results in this discussion, 
assuming one is willing to ignore the discrepancies in the definition of the rate- 
distortion region mentioned earlier. 



A Sum-Rate Achievability for the Binary Era- 
sure CEO Problem 

Showing that a particular rate-distortion vector is achievable using the Berger- 
Tung inner bound is mostly a matter of finding the proper "test channels" 
— > C/f for the encoders. To prove ©, we use binary erasure test channels 
that are identically distributed across the encoders. In this appendix and the 
next two. the notation is drawn from Section 13.21 



Lemma 1 For any < D < 1, 



lim 7^f ^(Z?, A) < (1 - D) log2 + L 

A — *-oo 



h (^I)!/-^) - {l-p)h 



P 



1-p 

Proof. Fix D and let iVi, . . . , be i.i.d., independent oi Yq, . . . ,Y]^, with 

D^/^-p 



Pr A^i = 



Pr nvi = 1 = 1 - 



1-p 
D^'^-p 



l-p 

For £ in {1, ... , L}, let Ui^Y^ Ni. Then let 

-1 iiEtiUe<0 
Z,=sgn[J2Ui] := \0 if ^ti Ue ^ 

otherwise. 
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Then for all A, 

E [d^{YQ,Y, Zi)] = Pr(t/f = for all I) = [Pr(C/i = 0)]^ = £>. 
Thus (R, D) is contained in UVf^ for all A if for all A C {1, . . . , L}, 

^i?, >/(Ya;Ua|UaO- (16) 

The rate vectors satisfying this collection of inequalities are known to form a 
contrapolymatroid |29l ■ As such, there exist rate vectors R satisfying H16|l 
such that 

L 

^i?, = /(Y;U). 

1=1 

In particular, this holds for any vertex of [^HllSni- Now 
/(Y;U) =/(yo,Y;U) 

= /(yo;U) + /(Y;U|ro) 

= I{Yq;V)+LI{Yi;Ui\Yq). 
But /(Fo; U) = (1 - D) log 2 and 

I{Y^-Ui\Yo) = H{Ui\Y^) - H{U,\Y,) 

Then for any A, there exist vectors (R, D) in TZVf^{X) such that 



^ i?£ = (1 - D) log 2 + L - (1 - p)h ( 



l-p 



=1 

The conclusion follows. □ 

B Sum-Rate Converse for the Binary Erasure 
CEO Problem 

We evaluate the outer bound's sum-rate constraint for the binary erasure CEO 
problem via a sequence of lemmas. Throughout this appendix, g{-) will denote 
the function on [p, oo) defined by 



Mx)-(l-p)Mf5f) P<x<l 
a; > 1. 



We begin by proving several facts about g{-). For this, the following calculations 
are useful. 
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Lemma 2 For all x in (logp, 0], 

log(e'' - p) - xe" < -p (17) 

and 



g2. 



e^log(e^-p)-e^(a; + l) + > 0. (18) 

— p 



Proof. It is well known that 

log f — ^ — I > ^ for all z < 1 . 

\l-zj 

Replacing z with pe~^ and rearranging yields p7|) . To see p8|l . note that p7|l 
implies that the first derivative of 

(e" - p) log(e" -p)- (e" - p)(2; + 1) + (19) 

is nonpositive on (logp, 0]. Since the function in 119|) is nonnegative at x = 0, 
it follows that 

(e" - p) log(e" -p)- {e^' - p){x + 1) + e^^ > (20) 

for all X in (logp, 0]. One can now obtain p8|l by multiplying both sides by 
and dividing both sides by (e^ — p)- O 

Lemma 3 The function g{e^) is nonincreasing and convex as a function of x 
on [logp, oo). 

Proof. The first derivative of g{e^) on (logp, 0) is 

e"" log(e^ ~p) ~ xe"^. 

This observation, the first conclusion of Lemma |21 and the continuity of g{-) 
together imply that .g(e^) is nonincreasing on [logp, 0]. Since g{e'^) is constant on 
[0, oo), it follows that ^(e^) is nonincreasing on [logp, oo). The second derivative 
of g(e^) on (logp, 0) is 

log(e'' - p) - e'^(a:; + 1) H . 

— p 

This observation, the second conclusion of Lemma |21 and the continuity of g{-) 
together imply that g(e^) is convex on [logp, 0]. Since g{e'^) is nonincreasing on 
[logp, oo) and constant on [0, oo), it follows that g{e^) is convex on [logp, oo). 

□ 

Corollary 1 The function g{y^^^) is nonincreasing and convex in y on [p^, oo). 

Proof. g{y^^^) ~ g{e-^) with x = [1/ L)\ogy, and g{e^) is convex and nonin- 
creasing while (l/i)log(-) is concave and nondecreasing. □ 
The next lemma is central to our evaluation of the outer bound's sum rate. 
Note that condition (i) in the hypothesis implies that Pr(yo • < 0) = 0. That 
is, the reproduction Zi is never in error (although it may be an erasure). 
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Lemma 4 Suppose < D and (U, Zi) is such that 

(i) E[d^{Yo, Zi)]<D for all X, 

(ii) Ui Yg (Yo , Y^c , Uf c ) for all £, and 

(iii) (lo, Y) ^ Zi. 
Then 



Proof. For each encoder £, let 

Ae^+ = {ueUi: Pr(C/^ = u\Yn = 1) > 0} 
^{u^Ur- Pr(f/£ = u\Yo = -1) > 0}. 



Then define 



Finally, let 



1 liUte Ai^+\Ai- 
Ui^{-1 if Ut e At^^\At^+ 
otherwise. 

6i^+ = Pr(;7£ = 0\Yt - 1) 

5£,_ = Pr([7£ = o|rf = -i). 



Then 



L L 

-^/(y,;t/,|ro) > -5]/(rf;t/,|yo) 

1 ^ 



is 

1=1 



i/i (p + (1 - p)(5£,+) + i/i (p + (1 - p)5t,-) 



-]-{l-p)h{5,^+)-]-{l-p)h{5,,^) 



Since Yq ■ Zi > a.s., on the event Zi = 1 we must have Yq = I and hence 
Ue G for all £. In addition, the condition Yq ^ U ^ Zi dictates that 

when Zi = 1 we must have Ug G for some £, for otherwise we would 

have Pr(yo • = ^1) > 0. All of this implies that sgn(^^^j^ IJe) = 1 on the 
event that Zi = 1. Similarly, sgn(^^^-^ IJi) ~ —1 on the event Zi = —1. Thus 



sgn(X]£=i = implies that Zi 
Pr l^sgn Ui 



0, so 

= I < Pr(Zi = 0) 



D. 
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This implies that 



L L 

^ \{{p + (1 - v)6l+) + X n(p + (1 - p)^^^-) ^ ^■ 



Thus 



- I{Yr, Ui\Yo) > inf <^ ;^ ^ 2 + " " " P)HSi,+) 



/i(p+(l-p),5,,_)-(l-p)/i(,5,,_) 



(5^,+, Jf,- e [0, 1] for all I and 
1 ^ 1 ^ 1 

2 + (1 - p)^^'+) + 2 n(^' + (1 - ?')^^-) ^ ^ ■ 

f=l f=l J 



This optimization problem is not convex, but if we change variables to 

= log(p+ (1 -p)5t^+) 
Ae^^ = log(p+ (1 -p)Se^-), 

then it can be rewritten as 



h{e^'+) - {l-p)h 



l-p 

+ h{e'^^^-)-{l-p)h[^-^ 
A£,+, A£._ e [logp, 0] for all t and 
i exp A,.+^ + i exp A,,_ j < 

A<^_+, A£ _ e [logp, 0] for all £ and 

which is convex by Lemma |21 Thus we may assume without loss of optimality 
that 

Ai,+ = A2,+ = --- = Al,+ =: A+ 
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and 

Ai.- = A2,_ = ••• = Al.- =: A_ 

This gives 



1 i fi 1 

- U,\Yo) > inf -g (e^+) + -g (e^-) : A+, A_ e [logp,0] : 

e=i [ 

> inf 1^ (e-^) : A G [logp,0] : e^"^ < L»| 

by Lemma 13 □ 
The quantity I{Yi;Ui\YQ) can be interpreted as the amount of information 
that the £th encoder sends about its observation noise^. Lemma 0] then says 
that if a fraction D of the output symbols is aUowed to be erased and no errors 
are allowed, then the amount of information that the average encoder must send 
about its observation noise is at least g{D^^^). We would like to extend this last 
assertion to allow "few" decoding errors instead of none. To this end, we will 
employ the following cardinality bound on the alphabet sizes of the auxiliary 
random variables Ui, . . . ,Ul- 

Lemma 5 Let (U, Zi) be sueh that 

(\) Ue^Ye'^ {Yo,Yia,Vec) for all i, and 

(n) (yo,Y)<->U<^Zi. 

Then for any X, there exist alternate random variables U and Z\ also satisfying 
(i) and (ii) such that 

E[d^iYo,Zi)]<E[di{Yo,Zi)], 
I{Yt, U,\Yq) - I{Yt, U,\Yq) for all £, 

and 

\Ui\ < \ye\ + l for alie. 

See Wyner and Ziv |H1 Theorem A2] or Csiszar and Korner 13 Theorem 3.4.6] for 
proofs of similar results. The next lemma is the desired extension of Lemma 01 

Lemma 6 Suppose < D and (U, Zi) is such that 
(i) E[dl{Yo,Zi)]<D, 

Ui ^ Yi ^ (Yq, Y^c, Ufc) for all £, and 

■'This terminology is due to Prabhakaran, Tse, and Ramchandran. 
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(in) (yo,Y) ^U^Zi. 
If 



l/L 

p{l^p)\\) -"-2 



then 

L 



J UAYo) > g ({D + 5)1/^) + 26 log | 



L 

1=1 



Proof. By Lemma |S| we may assume that Ut = { 1 , . . . , 4} for each £. We may 
also assume that Zi is a determiuistic function of U: Zi ~ <t>{V). Define 

Ai,+ = i ue £ U( : 3 Ufc : 0(u) = f and 



Pr(f/, = Ui\Yo = -1} = ^.^ min^^ Pr{U, = Uj\Yo = -f )| 
Ai^- = i^U£ ^ Ui : 3 Ufc : 0(u) = — 1 and 

Pi{Ue ^ Ui\Yo = 1) = min Py{U, = Uj\Yo = 1)\ . 

We now define random variables (U, Z) to replace (U, Z). The replacements 
will be close to the originals in distribution but will have the property that 
Pr(yi • Zi < 0) = 0. That is, Zi will never be in error. Set = {!,..., 5} for 
each £, and let 

f if i e Ai^^ 

Pi{Ui = i\Ye = 1) = S P^{Ui e Ae^_\Yg = 1) if i = 5 

[Pr(C/£ = i\Y£ = 1) otherwise 

'O ifieAi^+ 
Pr{Ue = i\Ye = -1) = <( Pr(C/£ € Ae^+\Yi = -1) if i = 5 

Pr(?7f — i\Yi — —1) otherwise 

{0 if i e Ae^+ U Ai_ 

PiiUe e Ai^+ U Ae^^ \Yi = 0) if i = 5 
Pr{Ue = i\Yi = 0) otherwise. 

Then define 

(j){u) iiui < A for aU £ 



Zi = 0(U) 



otherwise. 
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There is a natural way of coupling U to U such that if C/^ is in {1, . . . , 4} then 
Ui = Ui. With this coupling in mind, it is evident that 

E[d^{Yo, Zi)] = £;K(ro, ^i)l(max(C/i, ...,Ul)< 4)] 

+ ^K(ro, ^i)l(max(J7i, ...,Ul)= 5)] 

< E[4{Yo, Zi)l(max(J7i, ...,Ul)< 4)] 
+ Pr(max(;7i,...,J7L) = 5) 

<D + Pr(max([/i, ...,Ul)^ 5). 

Now for any £ in {1, . . . , L}, 

PriUe = 5) = i-^Pr([/, g = 1) + i-^ Pr(t/, G = -1) 

+ pPiiUe e Ae.+ U Ae^^\Ye = 0). 
By the union bound, this is upper bounded by 
"1-p 



Pr(c/, € = 1) +pPr(c/, e = o) 



i-p 



Fi{U( e = -l)+pPr{Ue G A^.+ IF^ - 0) 



Since Ui Yi ^ Yq, 



Pr{Ue = 5) < 



l-p 



FviU( e Ae,-\Ye = l,Yo ^ I) 



+ pPr(C/, e A,,_|y, = o,yo = i) 



But 



which implies 



+ pPTiUeeAi^+\Ye = 0,Yo^-l) 
< PiiUe G Ae^_\Yo = 1) + Pr(C/, e A,,+ |yo = 

1 ^ 

u:0(u) = l J = l 



on 

5] J|p,(C/^_^,^.|yo^_l)<_. 

u:0(u) = l,Mf eAf,+ J = l 



(21) 
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By the definition of , for each G there exists at least one u^c such 

that 0(u) = 1 and 

L 

Y[ Pr{Uj = u,\Yo - -1) > Pr{Ue = Ui\Yo = -1)^. 

Together with (|21|) . this imphes 

. , 2D 

^ Pt{U, = w.lFo = -1)^ < — • 

Applying Holder's inequality PHI P- 121] gives 
Pr(f/, e Af.+ |yo = -1) = MUi ^ Ui\Yo ^ ~1) 



< 



< 4 



A 



l/L 



\(L-1)/L 



Likewise, 



Thus 



Pr(f/^G A^,_|ro = l)<4 



'2D\ 

Pr(C/, = 5) < 8 ( — j . 



By the union bound, it follows that 

Pr(max(C/i,...,C/L) = 5) < 8L 



2D 



l/L 



and therefore 



2D 



l/L 



i;K(ro,Zi)] <D + 8L(^— j <D + <5. 

Note that Zi = 1 only if [/^ is in Ai,^ for some £, and 

Pr(C/f e ^f,+ |ro = -1) = for aU I. 

Thus Pr(yo = -1,^1 = 1) = and similarly, Pr(yo = l,^i = -1) = 0. It 
follows from Lemma 0] that 



, L 



(22) 
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The remainder of the proof is devoted to showing that I{Ye; UelYo) is close to 
I{Yf, Ui\Yo)- For this we use the decomposition 

I{Ye; Ui\Yo) = H{Ut\Yo) - H{Ut\Yt). 

Observe that 



l/L 

Pr(C/, = b\Yi = 0)p < Pr([/, = 5) < 



Thus 



Similarly, 



and 



Pr(U, - 5|y, = 0) < -. 



Pr{Ue = 5|y, = 1) < 2 



Pr(Ue = 5\Ye = -1) < -. 
Therefore if we view Ue as a random variable on {1, . . . , 5}, for any i in { — 1, 0, 1}, 

5 

I P<Ue = 3\Ye = i) - Pr((7, = j\Ye = i)\ = 2Pr((7, = 5\Y, = t) < 5. 

A standard result on the continuity of entropy Lemma 1.2.7] now implies 
that (recall (5 < 1/2) 

\H{Ue\Ye ^ i) - HiUe\Ye = i)\ < ~S log- 

5 

so 

\H{Ue\Ye)^H{Ue\Ye)\ < -Slog-. 

5 

Likewise, for any i in {—1, 1}, 

-Pr([/, = 5|yo = *)<8(^— j . 

Thus 

Pr({/, = 5|yo = *)< 2' 

so 

\H{Ut\Y^)-H{Ui\Yo)\<-5log- 
as before. It follows that 

\I{Yf,Ui\YQ)~I{Y,-Ui\Y^)\ < -2,5 log-. 

5 



31 



Combining this with (|22|l yields 



J J2 HYf, Ue\Yo) >g{iD + 5)1/^) + 2S log ^ 



□ 



Wc are now in a position to prove the main result of this Appendix. 
Lemma 7 For any < D, 

lim 'Ro{D,\) > (1 - D)log2 + i.g [d^/^' 

A— »oo \ J 

Proof. Fix < D <1 and 5 G (0, 1/2], and suppose A satisfies 

2L / 2" 

4 



A > max 



5p{l -p) 



(23) 
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l/L 



< s 



p{l-p)\ A 
by (|23l- Thus, by Lemmail if (w, i) € 5, 

1 ^ 



26 log-. 





By averaging over (w,t) G S and invoking Corollary^ we obtain 



(24) 



By taking X = Yq in the definition of TZVoiX), it follows that there exist R in 
M.^ and 7 in Fq such that 

IJ + J > £;[d^(yo,^i)], and 

L L 

7^o(-D, A) + 5 > ^ i?, > I{Yo; U\T) + ^ /(F,; Ue\Yo, W, T). 
1=1 1=1 

For eaeh possible realization {w^t) of (VF, T), let 

Let S = {(w,t) : -Diu,t < \/A}. Then by Markov's inequality, 

Pr((M^,r)^^)<^<,5. 
In particular, Pr((T^,T) e S") > 0. Also, for any {w,t) e S, 



(25) 



5((D + (5)i/^) + 251og 
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From it follows that 



Y,IiYi;Ui\Yo,W,T)>L{l-S) ,9 + 5)1/^) +2,5 log - 

£=1 L 



Now by the data processing inequality, 



/(ro;U|T) = /(yo;U,r) 



Let e = l(Fo ■ -Z^i = ^ !)• Continuing, 



/(>o;U|r)>F(yo)-i?(io|^i) 

= \og2-HiYo,e\Zi) 

= log2-Hie\Z,)^H{Y„\e,Z,) 

> log 2 - A) - Pr(Zi = 0) log 2 

> (1 - i:>)log2 - h{5). 



Substituting this and into lI23Jl yields 
Ro{D, A) > (1 - D) log 2 - h{S) + L(l - 5) 



(26) 



2,5 log- 



-S. 



The proof is terminated by letting A — > oo and then ,5^0. 



□ 



C The Berger-Tung Outer Bound is Loose for 
the Binary Erasure CEO Problem 

We will show numerically that for one instance of the binary erasure CEO 
problem, TZT?^^ contains points with a strictly superoptimal sum rate. Let 
L = 2 and p = 1/2. Let Wi and W2 be {0, l}-valued random variables with the 
joint distribution 

r 1/5 2/5 " 
2/5 J ' 

i.e., 

Py{Wi = 0, 1^2 = 0) = ^ 
5 

Pt(Wi = 1, 1^2 = 0) = Pr(VKi = 0, VK2 = 1) = -. 

5 

We assume that (M^i, VF2) is independent of {Yq, Yi,Y2). Let Ue ~ Yi ■ We for i 
in {1, 2}, and let Zi = sgn(f/i + 1/2)- Since Y^ can be written as Ye = Yq ■ Ne 
where A^i and N2 are i.i.d. with Pr(7Vi = 0) = Pr(7Vi = 1) = 1/2 (recall the 
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notation of Section l3.2\i . we have Ug = Yq ■ Ng ■ Wi. Note that A^i • Wi and 
N2 ■ W2 have the joint distribution 

" 3/5 1/5 " 
1/5 

Thus for any A, E[d^{Ya, Zi)] = Pr(A^i • W^i = TVs • VF2 = 0) = 3/5. Now we can 
compute 

/(Yi, Fa; Ui, U2) < 0.6273 nats 

and 

I{Yi , ^2 ; f/i IC/2) = I{Yi ,Y2;Ui,U2)- I{Yi , Ya ; C/2) 
< 0.3248 nats. 

It follows that (0.3248,0.3248,3/5) is in 7^X>f^(A) for any A. Thus 

^lim inf + R2 : (^Ri,R2, e TZV^'^ {X)^ < 0.6496 nats. 
From the previous two appendices, the correct sum rate is 

^1 - log2 + 2.9 (^\ffj ^ 0-6562 nats. 

D Evaluation of the Outer Bound for the Gaus- 
sian CEO Problem 

Two lemmas are needed for our proof of Proposition |3 The first is a simple 
extension of Theorem^to the Gaussian CEO problem setting of Section lTSI For 
this appendix, let us redefine x to be the set of real-valued random variables X 
such that Yi , . . . , are conditionally independent given X (the side information 
Yt+i is unneeded in this context and shall be ignored). Let us also redefine To 
to be the set of random variables {Ui, . . . ,Ul, Zi,W,T) such that each takes 
values in a finite-dimensional Euclidean space, and collectively they satisfy the 
Markov conditions defining the original Fq, 

(i) {W,T) is independent of (Yq, Y), 

(ii) Ue <^ {Ye, W, T) ^ (Yq, Y,o, U^c) for all I, and 
{ill) (Yo,Y,M/)^(U,T)^Zi, 

and one new technical condition, 

{iv) the conditional distribution of Ui given W and T is discrete for each I. 
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Note that any conditional distribution involving these random variables is well- 
defined 1^ Theorem 6.3]. As such, so is any conditional mutual information |21[ 
Ch. 3, especially the translator's notes at the end]. 

Lemma 8 For the Gaussian CEO problem, TifDi, C TifDo if Ti-'Do is defined 
using the x '^^'^ just described. 

The proof follows the original and is omitted. The second ingredient is 
a consequence of an intriguing result of Oohama |13j and Prabhakaran, Tse, 
and Ramchandran that relates information the encoders send about the hidden 
source to information they send about their observation "noise." 

Lemma 9 (c.f. [T^l Lemma 3]) // 7 is in To, then for all A C {1, . . . , L}, 

l-cM-2I{Yt,Ut\Yo,W,T)) 



exp(2/(ro;U^|W^,T)) < 1 + ;^ 



Proof. For any realization of {W, T) , it follows from Lemma 3 in Oohama [T5| 
that'^ 

exp(2/(ro; UaIVF ^w,T^t))< 

1 - cxp(-2/(y,; Ui\Yo, W = w,T^t)) 



1 + E 



We now average over {w,t) and invoke the convexity of cxp(-) twice, once on 
each side. □ 
Proof of Proposition\^ If (R, D) is in TZDo, then there exists 7 in To such 
that E[{Yo - Zxf\ < D and for aU A C {1, . . . , L}, 

J^Ri > HYo;Va\Va^,T) + J2l(Ye;Ui\Yo,W,T). (27) 
ieA e^A 

Now 

I{Yo; U^IU^e, T) + /(Fo; |r) - /(Fo; U|T). (28) 
Since Yq ^ (U,T) ^ Zi, the right-hand side can be lower bounded as follows 

/(yo;U|r) = /(yo;U,r) 

> I log 



2 " E[{Yo-Z,)^y 

where we have used the rate-distortion theorem for Gaussian sources [3 Theo- 
rem 13.3.2]. In particular, 

I{Yo;U\T)>^log^. (29) 



*Oohama's result assumes that Ui is a discrete, deterministic function of for each £, but 
the proof shows that conditions (ii) and (iv) above are actually sufficient. 
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Let us address the second term on the left-hand side of (I28II. Observe that 



I{Yo; Va^ \T) + I{Y„;W\Va^ , T) = I{Yo; W\T) + I{Yo; Va^\W, T) 

= I{Yo-\JaAW,T). 

Defining ri = I{Yi]Ui\Yi^,W,T) and applying Lemma|51to the right-hand side 
gives 



I{Y^-\5AAT)<Uog 



1+E 



l^A" 



1 — exp(— 2rf) 



(30) 



Substituting and (jSOj) into gives 



/(yo;U^|U^=,r)> -log+<( - 



1 - cxp(-2r£) 



The conclusion follows upon substitution of this inequality and the definition of 
ri into (123). □ 



E The Berger-Tung Outer Bound is Loose for 
the Gaussian CEO Problem 

We have just seen that the improved outer bound is capable of recovering the 
converse result of Oohama [TJ] and Prabhakaran, Tse, and Ramchandran 
for the Gaussian CEO problem. Here we will show that the Berger-Tung outer 
bound does not recover this result. As with the binary erasure CEO problem, 
we will show that, in general, the Berger-Tung outer bound contains points with 
a strictly superoptimal sum rate. 

Consider the case in which, in the notation of Section 13.31 L = 2 and tr^ = 
(7^ = (t| = 1. In words, two encoders each observe a unit variance, i.i.d. Gaussian 
process in additive Gaussian noise with a signal-to-noise ratio of unity. It follows 
from Proposition|51that the minimum sum rate needed to achieve the distortion 
1/2 is at least (3/2) log 2 nats. 

Let W , Vi, and V2 be Gaussian random variables, independent of each other 
and Yq, Yi, and Y2. Let Vi and V2 have unit variance; we denote the variance 
of W by cr^ . Let 

Ui=Yi + Vi+W 

U2^Y2 + V2- W. 

Note that the sum of Ui and U2 is a. sufficient statistic for Yq given Ui and 
1/2- This observation makes it easy to verify that if Zi = E[Yo\Ui,U2], then 
E[{Yq — Zi)'^] = 1/2. Note that this distortion is independent of a^. 

It follows that for any value of a^, TZVj^^ contains points of the form 
(i?i,i?2,l/2) with 

Ri + R2= max(/(ri. Fa; C/i, C/2), 2 /(Fi, 1^2; Ui\U2)). 
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But d Theorem 9.4.1] 

m,Y2;U,,U2) = -10,2 + -lo,^ 



3, „ 1, 1 + crl 



' w 
and 



I(Y, ,Y2;Ui\U2)^ I{Y, ,Y2;Ui,U2) - I{Y,,Y2;U2) 
= I{Y,,Y2;Ui,U2)-I{Y2;U2) 

= -log2+-log ^ + -llogfl + ^j- 

Observe that, when viewed as functions of ct^, I{Yi,Y2;Ui,U2) is strictly de- 
creasing and /(Yi, Ui\U2) is continuous. Since cr^ = yields 

j(yi,y2;;7i,c/2) = ^iog2 

2I{Y,,Y2;Ui\U2) < ^log2, 
it follows that there exists cr^ > such that 

msix{I(Yi,Y2;Ui,U2),2I{Yi,Y2;Ui\U2)) < ^log2. 

F nvf^ is Closed 

The main step in proving that TZDf" is closed is to show that one can limit the 
ranges of the auxiliary random variables without reducing the region. 

Definition 5 Let Vf^ denote the set of finite- alphabet random variables 

7= {Ui,...,Ul,Zi,...,Zk,T) 

in Tf" sueh that 

\Ui\ = \yi\+2^ + K - \ for all £ 

and 

\T\ = 2^ + K. 

Then let 

Tivf^ U 7^pr(7). 



We shall show that TZTyf^ ~ T^T^i in two steps, first handling the case 
in which T is deterministic, and then bootstrapping to the general case. Both 
steps involve now-standard uses of Caratheodory's theorem Theorem 17.1]. 
We give proofs of both steps, albeit condensed ones, due to the complexity of 
our setup. 
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Lemma 10 Suppose that 7 = (U, Z, T) is in TZV^ and T is deterministic. 
Then there exists 7 = (U,Z,r) in TifD^ such that T is deterministic and 

Proof. For any Ad {1, . . . , L} containing 1, we have 

I{Ya;Va\Ua^) = H{Ya\Ua^) ^ H{Ya\U) 

= i/(Y^|U^c) - ^i/(YA|Uic,C/i wi)Pr((7i = ui), 

Ml 

while for any nonempty A not containing 1, we have 

/(Ya;U^|U^c) = ^/(Ya;Ua|U^c\{i},C/i = wi)Pr([/i = 

Ml 

Carathcodory's theorem guarantees that we can find a. Ui with Ui C Ui such 
that \Ui\ = \yi\+2^ +K -1, 

Pi{Yi =yi\Ui^ui) Pr(J7i = Ml) = Pr(Yi = yi) for all yi in but one, 

Ml GWi 

^ H{Ya\Ui^,Ui=ui)Py{Ui=ui) = 

MiGWi 

iJ(YA|Uic, fJi = Ml) Pr(J7i ui) for all A containing 1, 

Ml eUl 

and similarly for /(Y^; Ua|U^<:\{i}, ?7i ui) and i;[dfc(ro, Y, Y^+i, Zfc)|C/i = 
Ml]. Since C/i <-> Yi ^ (Fo, Yic , Uic, Yj^^i), if we substitute Ui for [/i, the 
resulting 7 is in Pf ^ and 7?,I?f "^(7) is unchanged. Repeating this procedure for 
U2, . . . ,Ul completes the proof. □ 

Lemma 11 7^2?f'^ = 7^X>f ^. 

Ptoo/. Let (U, Z, T) be in Pf^. For each t in T, let (U, Z, i) denote the joint 
distribution of (U, Z,T) conditioned on the event {T = t}. By Lemma [TUl for 
each t, there exists (U,Z) such that (U, Z,t) is in ff^ and TZVf^ (U ,Z,t) = 
nvf^{\5, Z, i). By replacing (U, Z) with (U, Z) for each value of T, we obtain 
(U, Z, T) in Pf ^ such that \Ui\ ^ \yt\+2^+K-l for all £ and 7^^»f ^(U, Z, T) = 
nVf^it.Z.T). Now 

7^Pf ^(U, Z, T) = I (R, D) : ^ i?, > 

^ /(Ya; Ua|Ua., y^+i, T = t) Pr(T = t) for all A, and 

Dk > E[dk{Yo, Y, Yl+1, Zk)\T = t] Pr(T = t) for all k \ . 
ter J 
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Caratheodory's theorem implies that we can find a T with T C T and |T| = 
2^ + K such that 

^(Ya; Va\Va^,Yl+i,T = t) Pr(T = t) 

ter 

= ^ /(Ya; UaIU^c, yL+i, T = t) Pr(f - for all A 
and similarly for E[dk{YQ, Y, yL+i, 2'fe)|r = t]. Then (U, Z, f) is in ff^ and 

7^I?f ^(u, z, T) = 7^I?f'^(u, z, f ). 

Since (U, Z,T) in Tf^ was arbitrary, it holds TlVf^ C 7^pf^. This completes 
the proof since the reverse containment is obvious. □ 
The cardinality bounds provided by the last two lemmas, while finite, are 
exponential in L and hence impractical for moderate numbers of encoders. One 
can improve upon these bounds by exploiting the polymatroid structure |29U26| 
of TZVf^. While this would be useful if one wished to numerically evaluate the 
bound, our aim here is merely to show that it is closed. 

~ BT 

Lemma 12 TZT)^ is dosed. 

Proof. The Markov conditions defining can be expressed as 

I{T;Yo,Y,Yl+i)^0 
I(Ut- Yq, Y,c, Fl+i, U^c |r,, T) = for aU £ 
/(ro,Y;Z|U,yL+i,T) = 0. 

Since the conditional mutual information function is continuous, F^"^ is compact 
when viewed as a subset of Euclidean space. Thus if (R'"', D^")) is a sequence 

in li'D^ that converges to (R, D), by considering subsequences we may assume 

that (R("), D(")) is in 7^X'f ^(7(")) for each n and 7^") ^ 7 = (U, Z, T) e ff^. 
By invoking the continuity of mutual information once again, we obtain 

Y,R,>I{Ya-.\5a\^m-,T) 

for each A. Likewise, 

Dk > E[dk{Yo,Y,YL+i,Zk)] for all k. 

It follows that (R, D) is in 7^X>f ^(7) and therefore also in TZV^ . □ 
Corollary 2 UVf^ is closed. 
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